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ABSTRACT 

\ Institutional modeling and program evaluation in 

relation to a correspondence program are discussed* The" evaluation 
process is first considered from the viewpoint that it is an add-dn 
activity, which is largely sumotative, and is the least desirable type 
of evaluation* Formative evaluation is next considered as a part cf 
^e^iprb^ the f irsft step, which 

continues as an ongoing activity for the life of the institution, and 
is thus the most desirable evaluation process* . Given the 
institutional concept and rationale for a new approach to the 
education of distant learners, including the over-all objectives to 
eb^: achieved, a model of the new institution is projected* A 
Conceptual matrix for building the correspondence institution is 
provided, which is useful in modeling because it (1) identifies all 
£Jie elements that must be dealt with, and (2) provides a way for 
builders to "select the universe" that they wish to work within* A 
tentative three-stage model, in which the learner roles become %he 
active focus, is provided* A planning matrix for modeling a 
correspondence teaching and learning institution is presented, and an 
appendix provides a list of the quality control subsystems that may 
be useful in fleshing* out a model for a correspondence institution* 
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fN- 1.0. Evaluation as an add-on activity 

^ 1.1. Evaluation is frequently thought: of as an add-on activity, after 

Q / • ' - 

the institution has completed its work. The rationale for evalua- 
tion as an add-on to operations proceeds this, way: 
L» First we planned our work on the basis of needs which 

were given to us as our responsibility to meet., — * 
2. Then we created the process to accomplish our work. ; # 

^ 3.^We-|^^ r * ---- - - 

7 . 4. We did our work.; '-}_-. - : ^ • ''. 

5. Now we_ must report to our superiors what we have accom- 
plished, and ask for; more; subsidy so we can continue 
- and grow. . - : •. V ; . , " _ " 

V; ; (add on; . -; • - - : ' 1 • . • 5 

.. ~ 6. What have Wv accomplished? ; . * 

1.2. As an add-on evaluation may be useful (and better than no evalna- 
tion at all)but the rationale is faulty and the consequences are 
hazardous. Consider, for example, the numerous examples you can 
• think of whereir program and institutional evaluation never occurs at 

ali* When you ask about evaluation, people hang their heads, look wis t- 
fully resolute and re^aark, "Yes. Well, we're going to do that later- 
just as scon as we get the time and the funds to do it. We've been so 
busy just operating the program we really. haven't had a chance to get 
started. But we intend to." (Rustle of papers.) '^ou know, I even 

made some notes on it once. Let's see. Maybe they've been filed some- 
's. - . ,-. . -i ■=■---' ." . • - 

where. . . " The hazard, of course, is that the five stage activity 

cited above is wholly operational; it does not include evaluation ex- 
cept, alas, as an^add-on, an afterthought. Furthermore, personnel com- 
mitted to operational stages that do not include evaluation, develop 

%. ^ ' ' - • K.}:y ■ - - 
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a growing sense of anxiety about wfi&t they are accomplishing. If • 
this anxiety is not reduced by actually doing an evaluation, it in- 
creases until it becomes itself a barrier to evaluation. Since the 
person does not really know what has been accomplished, the anxiety 
becomes fear of what may be revealed by a true evaluation. Even when 
such anxieties -cum- fears "are groundless (as they usually are) they de- 
finitely inhibit the undertaking of an evaluation. What is usually 
required to break ^this J>arrier is a crisis of some sort: 

1. an order from a superior — ; . _ 

2. a new top administrator who wants to make it clear that ' 
he's not responsible for what went on before he took over, 
requiring an evaluation to stake out the grounds and goals 

of his new administration 

3. problems o£ such magnitude that the survival of the opera- 
^_?=^£g^£jj T ;thf^ateiied^ " ^ "_" *_ ^ ^: : ~~ ----- -=_- --- - ----- 

4. an opportunity , for the program to grow dramatically if a case 

can be made on the basis of past performance. 4 
Evaluation as an add-on is largely summative; that is, it is an activity 
undertaken at some point to set forth what has happened and what has been 
accomplished up to that point. It may be historical or sociological in 
focus; it may be narrowly directed towards r management , cos t-benefit ra- 
tios, completion studies or other aspects; orit may be comprehensive in 
scope. An evaluation, however, is more than a summary of events or sta- 
tistics; itis a careful appraisal of events and statistics leading, to ^ 
the formation of judgements respecting what has happened (good-bad, use- 
ful -not useful, author! zed- not authorized) the identification of trends 
and the elaboration of cause-effect relationships . An .evaluation seeks 
the why as well as the what and how of things,/ and may well include re- 
commendations foLfuture action. It is highly analytical in nature , and 
depends not only on records of events, happenings and statistics, but al- 
so on generally accepted standards and criteria drawn from- the professions 
Comparative studies may be an important part of an evaluation if the per- " 
sons concerned wish to know how the activity evaluated compares with 
similar activities conducted elsewhere. 



Suramative or add-on evaluations are generally carried out by per* 
son* who-are-separate fronts and neutral- towards the activity eyaluT 
ated. Objectivity on the part of the e valuators is essential. If 
the manager or staff of the activity conducts a suramative evaluation, 
there is conflict of interest, and the evaluation is suspect* 
Sunmative evaluations by separate and neutral persons ought to be 
carried out: periodically (at least every three years) on a planned 
basis, not a mere add-on under the stress of a crisis which may have 
arisen, in part, because there were no periodic evaluations to reveal 
what is happening, why, and to provide considered courses of remedial 
or developmental action,- on the basis of accepted criteria. 
Evaluation as a Function of Institution Building 

If evaluation is conceived of only as add-on or suramative, what does 
the new institution or program do to find out what is happening, why, 
and to make decisions for- remediation t ot development?- Does one have 
^to wait for a crisis^ or for planned suramative evaluation after a period 
of operation? Is there * process by. which the advantages of evaluation 
can be continually ava? le during institutional development? The con-,: 
cej>t of . continuing eval*. v ion In institution building is important. It 
is, for example, an essential part of what has been called "systems de- 
sign," a process that has evolved specifically to guide project and in- 
stitutional development • As such, evaluation becomes a function of in- 
stitution building itself, organic and integral to 'development, not add- 
on or suramative. In this context evaluation is sometimes called "forma- 
tive" or 

Formative evaluation is a part of the process of institution building from 
the first step, and continues as an ongoing activity foj: the life of the 
institution. It does not replace suramative evaluation, which should oc- 
cur periodically, and which is separate and neutral. Formative evalua- 
tion Is carried on by the regular staff; it is not neutral and not sepa- 
rate. The people who use formative evaluation are committed to the mis- 
sion and objectives of the institution because they have participated in 
their development; they are responsible for building the institution, its 
programs and services, and they are thus accountable for the consequences 



of design, strate^, , programs, processes and ultimate achieve- 
ment. Responsibility and accountability require formative evalua- 
-tion both for the self -interest and security-of the people in the - 
program, and for the survival of the institution itself. 
1.8. Formative evaluation, viewed in this way, is a function of insti- 
tution building. If employed as suggested, it can bring to insti- 
tution building the validity and vitality- that the emphasis on be- 
havioral objectives" has brought to teaching in the past -few years. 
Formative evaluation in institution building is creative, formative. 
The institution being built is the outcome of a complex of creative 
processes which derive their validity — always-- from mission, objec- 
cives, goals. This kind of evaluation continually throws inquiry 
respecting what is going on back to original questions: 

What are we committed to do? (Mission, objectives, goals) 
Why are we doing these things? (Social needs rationale) 
How-best^can: we? accomplish what Ve^are/coiindtte^ -itf^do?' 
\ : ^ (^ttod,' process) ^i/ v - V : -~ : v " - ^ - 

Is what is actually happening consonant with our mission, 
x ; objectives, goals? (Responsibility) * 

Why or"why not? (accountability) : _ j 

What decisions must we make now to improve the institu- 
- m tion? ^ ^(continuing development, building) 

Institutional Concept and Rationale - v. 

2.1. Formative or developmental; evaluation is organic and integral to the 
building (and operating) of the institution. Consequently, formative 
evaluation begins with the conceptualizing of the institution or pro- 
gram. In conceptualizing the program a theoretical model is being 
created, a prototype of the institution which will come to be through 
successive stages of development. The model is useful in making con-r 
crete and realistic the conceptual abstractions of institution building, 
and in guiding the development of the' actual institution. - • 

2.2. The general concepts that precede the formation of the institution give 
the answers (or the direction in which to search for answers) to such 
basic questions as: * - 















Why is the institution needed? How were these 
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~-7 needs determined? -Arei they valid? 7 








What is the authority for Tt1»rin^tituV£n?.7~ 77 
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Who is the institution to ssrve? 
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Where are the learners; what is their condition 








and situation? 77 - . 
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What educational services shall be provided? A 

How well do the services match with the 
- needs? — • 
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"~"\*Who'id."li developTthese program services? ,C * 
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How will the services be brought to the learners? 
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What outcomes are expected? 
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If you look back over these questions you will see that they con- 








stitute an outline of, and rationale for, the institutional concept. 
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2.3. 


The Institutional Model and Evaluation 
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2.4. 


Given the iris titutional concept and rationale for a new approach to 
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the education of distant learners, including the overall objectives , 
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to be achieved, a model of the new institution can be projected. The 








model must define the "universe" within which the institution exists. 
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All the elements which are nbw^part of reality must be present in the 






model system. In addition, as a response to the tensions inherent in 








the acceptance 6f objectives, the model system must picture the dyna- 
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mics of activity, of movement, of things being done to achieve the ob-7 
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jectives, to reduce the. tensions through successful achievement. HenSe 
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the model system has to introduce these new elements, activities, struc- 
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tures, measures* arrangements that are hot jpart of the present reali- 
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ties* A model system, therefore, will include elements that are hovel 
or unique at the model building stage. These uniquenesses, as soon as 
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implemented in practice, become part of the realities of the system. 
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So a model must cope not only with things as they are, but also with 
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things as they will be. 
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2.5. 


A model system is also characterized by subparts, or subsystems. The 








subsystem (or submodel) must be as viable to its purposes as the overall 
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model, sysjtem. Each part of the system and subsystems must be tested' for 








validity in every way known beforehand as well as during its operation. 
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Any, system implies* a condition of equilibrium; that is, given the con- 
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ditions, the objectives, the forces acting, the activities, there should 
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be certain predictable, outcomes. A change in any part of the 
system produces change in the other parts of the system. 

Mas^r i(o human system actually operates so precisely . 
There may be errors in^ defining the 'hihi verse! 9 within which the 
system operates because of unknown or*unperceived elements which 
are not taken into account* There may be changes introduced un- 
knowingly at some point so that adjustments to other parts of the 
system are not made^concurrently • Predictability thus diminishes. 
Reliability falters. Validity fades. /* c — 

That-is why thfe 'ro del system suggested here has a built-in forma- 
tive evaluation or ^sensing element . If feedback warns the insti- 
tution builders of something unanticipated, of a failure at any 
point, an immediate analysis of the system model in situ is called 
for. The equilibrium of the system which makes it reliable, and 
outcomes reasonably predictable; must be restored. Consequently 
an evaluation "scheme which only comes into being to measure effect 
after institution building is an extravagance that human planners 
who care about their work cannot afford* Even success, evaluated 
after the fact* offers little enlightenment, institution builders 
use forma tiye Revaluation to analyze what is going on, almost day- 
to-dayr, make, decisions on the basis of known realities, and continu- 
ally develop and refine the model system until it does achieve that 
reasonable equilibrium that assures reliability and predictability. 
Such ah evaluation does hot* require control groups, since its purpose 
is not comparison^ with some presumed standard. However, operation 
of the model will eventually produce norms or standards which 
can^be used in other kinds * of evaluation or as part of • periodic sum-/ 
mative evaluations. (2) / ; ^ ; 

Building an institutional model* is therefore not a simple matter of 
putting together a project according to a pattern or template. It is 
a creative effort to achieve the thrust and xounter thrust, the tension 
and counter-tension, the actions and reactions that^the model systems 
hypothesize are needed to produce the results desired. Arid since in- 
stitution building is not carried on to prove or disprove, the models, 
but rather to make advances in human organization and activity towards 



better education, in this case of distant learners, the burden of 
the institution builders is to work creatively with the system to 
make it produce what is needed. Continuing' evaluative feedback 
will make that kind of development possible, and will also contri- 
bute to later summative evaluation, (3) To present. formative evalua- 
tion as a function of institution building it is necessary to pre- 
sent a view of institution building. Hence the succeeding pages 
have a dual purpose: 1) a compact delineation of institution build- 
ing as a process, and 2) an exposition of the role of formative eval- 
uation within that process. While the presentation has general 
validity, it is specifically focused on the problems inherent in 
building a correspondence or independent study institution serving 
distant learners, (4) 

A Conceptual Matrix for Building the Correspondence, Institution 
The matrix presented* on page 8 cites the various situation realities 
that must be considered in constructing and operating a correspon- 
dence institution. Each of the realities is, in fact, a variable, 
because in no specific place or region where such a program is esta- 
blished will- the situation be exactly the same. The matrix is useful 
in modeling because <1) it .identifies all the elements that must be 
dealt with, and (2) It provides a way for builders to "select the 
universe" that they wish to work within, for example, institution 
builders may wish to approach their test universe through the selec- 
tion of target populations, through, a subject or content focus, or 
through some other situational reality. The matrix enables the 
builder-designer to locate the approach that is closest- .-to his situa- 
tion— that iits his resources, the population that he regularly works 
witlv, etc. He can select the universe that' is indigenous to him and 
thereby apply the special talent, expertise, resources and experience 
that his situation offers. 



The conceptual matrix suggested is: not complete; Each institu- 
tion builder may insert into the appropriate columns his mission, 
objectives and the other items that describe his universe,' and. 
delete what is inappropriate; to complete the conceptual matrix 
for his own institution. Vertical columns should itemize all 
the development concepts of the particular categories, spellei 
out in specifics. The first column (far left) will spell out 
mission and objectives. Hence by starting with a specific ob- 
jective on the left, it is possible to follow laterally through 
the succeeding columns to conceptualize the model ot system that 
is being built to achieve that objective. Of course it is not 
possible to get the grand design on one sheet of paper, as pre- 
sented here. Laying out the entire model or design in detail, 
unique to a specific situation, may requirean entire wail in 
your* off ice^ 

Laying out the matrix may look like a boring chore. It isn't. 
It's an exciting, intensive effort, for here before your eyes and 
under your hands the institution begins to come into being. The 
matrix is dynamic, not static. It changes as your conceptualiza- 
tion matures, as the realities of your universe are perceived or 
altered, as mission and objectives are modified. If nothing else 
the matrix is useful as a. series of check lists; but it will be 
more than that if it is employed thoughtfc'ly. 
The conceptual matrix is not at this point a system, model or in- 
stitution. The concepts fashioned in the matrix, however, are the 
basic ingredients of the next steps, modeling, testing and operating 
The modeling process suggested here consists of three essential 
elements— a systems development plan, an analysis of learner roles, 
and a matrix for modeling. •* 

Systems Development Plan . A systems development plan merely ite- 
mizes, in appropriate sequential order, the major task areas that 
must be completed in working towards a model system. 

Task I. Concept ' 
• Task 2. Mission, objectives, goals 
Task 3. Learner roles and needs 
Task 4. Resources required to achieve objectives 
with learners C»* 



Task 5* Communications media to be used^ 
Task 6. Coordinations required 
Task 7. Learner services 

Task 8. Curriculum, courses, diplomas, degrees 
Task 9. Continuing formative evaluation j 

Task 10* Staff required, regular and ad hoc 

Task 11* Organization and structure 

Task 12. Software and hardware 
~ Task 13*. Training, staff and learners 

Task 14. Finance, fees and budget (5) 
The order in which tasks are taken up is important. The aim of 
a systems development plan is to go from the concepts and genera- 
lisations of the conceptual matrix to quite specific plans that 
will be needed in the later stages, and to ensure that particular 
tasks (such as organisation and structure, or finance and budget* 
ing) are not considered until the .institution builders know in 



considerable detail what the institution will be so that practi- 
cal models that fit the requirements can be produced. 
Modeling to Fit Learner Roles . To construct a model, test arid 
operate it as an institution, it in now necessary to conceive, 
picture or visualize events as they are likely to occur in the uni- 
verse described by the conceptual matrix. The institution builder 
"acts out 11 the events intellectually, and puts them on pailer as a 
model of the institution and its behavior j. Modeling is thus a test 
of the completeness and appropriateness of the concepts first sketch 
ily drawn together in the matrix. It's not unusual to find that con 
cepts which looked adequate are too fuzzy to act put; or that ideas 
which seemed realistic at the conceptual level are at odds with cer- 
tain realities of the universe which are unchangeable givens in the 
situation; or that some concepts are in conflict with others. Hence 
modeling tests and improves the validity of the concepts upon which 
the institution is built. This is formative evaluation at work fo- 
cusing inquiry on concepts and processes, and providing feedback to 
the institution builder of what is complete, what works; what is 
incomplete, what doesn f t work; indicating voids in design and pro- 
cess, signalling the neee! for decision making, modification or re- 
design. ~ 
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Seven Learner Roles . The focus of the correspondence institu- 
tion it on the distant learner. If we look at the distant 
learner in his situation with the purpose of achieving the 
overall goals which have been stated on the matrix, we see that 
there are seven identifiable roles which the learner plays in 
relationship to the institution being built. " 

Role 1 - The learner is passive with respect to learn- 
ing because he thin?& he is learned enough to 

v survive and perceives no new, learning needs. 

Role 2 - He is anxious because he r.binka or fears that 
maybe he doesn't know enough , and begins to 
weigh whether and how he could learn general or 
specific things that would meet his needs bet- 
ter. His* needs are only vaguely perceived but 
^ ~ "-"- = " tie is beginning to display ^oal seeking beha- 
viors • * 

Role 3 - He casts about \or leads which will put him in 
touch with learning opportunities to satisfy 
his needs in his situation. His needs are now 
more sharply perceived, and are being transmuted 
into goals. Anxiety increases, particularly if 
he fails to locate opportunity that is accessible 
to him. 

Role 4 - He acts on his goals, makes decisions among tUe 
possibilities open to him. He does something to 
enhance Kis learning, such as enroll in a learn- 
ing program. Whatever overt or covert action is 
taken to initiate purposeful learning, goals con* 
tinue toundergo modification* If the action 
[taken is formal, goals are modified according to 
institutional programs and accessibility. The 
learner displays learning or knowledge seeking 
behaviors . 



Role 5 - He becomes a student in a specific program. 
*- He begins learning* 

Role 6 - He persists (or does not persist) in learn- 
ing. 

Role 7 - He reaches (or does not reach) his and/ or 

the institution's goals. Anxiety is reduced 
if successful; increased if unsuccessful. Fur- 
ther goal modification. 
These seven goals are cyclical. For example^ In -playing out Role 
7, the learner whose needs and goals are in equilibrium will cease 
formal learning and return to Role 1. Or, if needs and goals are 
not in equilibrium, he will go back to Role 2 or 3. Similarly 
success or failure (for whatever reason) in Roles 3, 4, 5 and 6 
will push the learner i ahead to the next role, or will force him 
back" to a preceeding role. Role 1 is characterized by passivity, 
which, as needs are perceived and transmuted into goals, gradual- 
ly phases into activity, first through goal seeking behaviors, and 
then through learning or knowledge seeking behaviors. The extreme^ 
dependency of the early roles is replaced (if the learning is suc- 
cessful and the institution 's^goals are met) by growing autonomy 
and independence in the later roles. If the cycle of roles descri- 
bed above is reasonably accurate for most learners (differences in 
socio-economic, cultural and educational background, geographic and 
situational circumstances naturally affect role behaviors) then the 
educational institution ought to focus on these roles* 
Institutional modeling has generally focused on institutional roles, 
not on learner roles. If the institution has the mission simply of 
teaching learners who are required to attend school (the source of 
most of our concepts of teaching, learning and the institutional mo- 
dels that hav§ evolved) the number and nature of learner roles per- 
ceived is somewhat different. Here the institution builder starts 
with the 'learner as a given. He works out a model according to what 
the institution has to dp from that point oit, with captive learners. 
Itiere is thus a tendency for the institution to become authoritarian 
to deal with the learner as a supplicant, to fashion objectives, 
courses, and processes in the image of the Institution and at its 
convenience. 



The institution which seeks to encourage self-selected, non- 
captive distant learners must develop its model on other con- 
cepts and on the roles played by its learners, this is not an 
easy task if the distant learner program is a department or 
division within an otherwise conventional institution. In such 
cases the institution builder has to face pressures, and a cli— v 
mate of educational theory and operating practice, that may be 
inimical to the concepts, purposes, roles and processes that he 
is trying to model. Nevertheless the builder of a correspon- 
dence or other types of mediated program for distant, indepen- 
dent learners has no real choice. If he wants to build to serve 
those learners, he must be prepared in his modeling to deviate * 
from the conventional. ^ 

A Tentative ^Three-Stage Model . From the conceptual matrix, sy- 
stems ^development plan, and the analysis of learner roles a ten- 
tative system model : can be constructed to accomplish the purposes 
in the universe and with the populations a resources that have 
been identified. The learner roles become the active focus of the 
, model; that is, the_institution is built around the actual roles 
of its 4 learners in order that it will seive learners at every 
stage of relationship and development. A model built around learner 
roles can be expressed in three/ stages. The exact tiumber is im- 
matei* ;p long as the subsystems accurately relate to the learner 
and tL >.es that he plays. These stages are cyclical, but they 
may be shown as linear or branched if that is preferred. 



/f Public Relations 
Recruiting * 
Counselling 
Goal Setting t\ 
Enrollment 
\ arnin 
Examining ^ * StageU. fS"^c?ion 

Certifying w 



Accrediting 
Trans fering 
Counselling 
Goal Adjustment 
Employment 



^ - # 2.19. Stage I * 
A subsystem designed to reach and serve the . 
learner in Role 1, in which the learner is pas- 
sive, and assumes that he knows what is neces- 
sary to survive, meet his needs and goals. 

There is lack of perception of the cotermin- 

- -- - . - - - . - - * 

ous relationship of ignorance and knowledge and 

. _ - - - / o 

there is lack of goal seeking and knowledge seek- 
ing behaviors. The learner (or, from the point 
view of the institution, the potential learner) 
may be neutral 'to .the .institution; or friendly or 
hostile, but that he needs .a relationship to the 
institution ^or learning* is* inconceivable to him. 

- The subsystem to cope with Role 1 learners] 
is sometimes called Public Relation 
ing. v^The purpose is to reiach as inany Role 1 ^ 
learners as pps sible , an d help; them advaric o through 
Roles 2, 3 and 4. Although Role 1 learners may 

-_ _ _ ^ _ __---- _ - - y _ __ __ 

seek to represent an undifferentiated mass of people 
the subsystem to reach and serve them must be discr* 
minatory. Mass recruiting is generally unsatisfac- 
tory, as is mass Public Relations. The subsystem 
must be rifled to reach those who are most likely 
to; respond: to the opportunities/ made accessible. 
Conmunications formats must be of the kind to which 
such learners pay attention and respond. Although 
the long range institutional and social goal may be 
to encourage purposeful learning through an enroll- 
ment in a. course, the Stage I enabling goal is ir^re 
general: to get the person to examine situation, 
needs and goals, and discover that continued learn- 
ing may contribute to the improvement of his and 
society's well being. 

Since it is well known that the more education 
one has, the more one perceives continuing needs for 
learning, the Stage I subsystem tends to build on 
what educational experience or awareness is already 



present among Role 1 learners* . 

Formative evaluation at this point fecuses 
on what happens, the successes and failures, of 
this stage of the, institution's model ^activity. 
The institution builders ought to be able to esti- 
mate how many Role 1 learners must be reached in 
order to get a yield of Role 2 learners, and so 
on. Demographic data as well as program data are 
needed. Who i£ to do these important tasks, what 
reports are needed, how often, and how they relate 
to decision making all along the line roust be care- 
fully built into th$ subsystems. 

/ The Stage I subsystem will also reach and serve 
Rble 2, 3 and 4 learners, notonly as they result 
frcwn ef forts with Role-1 learners, but also because 
the model activity will reach persons already in - 
Roles 2 and 3. 

;^ The subsystem hence will have specific enabl- 
ing objectives to guide activities with Role 2, 3 
and 4 learners % At some- point the general public 
relations and recruifment type activities will be 
replaced, for specific learners, with advising and 
counseling activities. Here again the institution , 
builder must be able to estimate the number of 
learners in each role that will yield, through the 
institutionVs model activities, the learners re- 
quired in successive stages so that the total sy- 
stem remains economically and academically viable. 
Formative evaluation at each stage and substage will 
analyse, pinpoint and critique in order to achieve 
the necessary flow of learners, and to generate the 
demographic data that are needed in order to design 
programs that will work. 

~2v2(h Stage II 
A subsystem that works with a learner who has enrol- 
led in the learning program. Stage II, is composed 
of two subsystems to accommodate Role 5 and 6 learners. 

Role 5 - the learner begins learning or 
does not begin 

Role 6 the learner persists or does not 
persist 



Each of these substages is designed to accomrao- 
date branching. Essentially, branching in the 
system proposed is a reversal of progress to a 
previous learner role/ It is important to note 
that if, for example, a Student who has progressed 
from Role 4 to Role 5 does not begin learning (the 
performance of learning tasks and the submission of 
evidence of learning) there may be many reasons, in- 
cluding a decision just to drop the whole thing; 
Perhaps 107 o of the? self selected, non-captive dis- 
tant, learners who progress to Role 5 (are enrolled, 
obtain materials, pay fees and exhibit, initial learn- 
ing seeking behaviors) actually do not 'begin. They* 
branch out, "in effect, before starting. Ten per- 
cent of an enrollment jls a high figure, and both for 
the learners: and the institution it is eminently 
worth while to have /a subsystem to cope with this 
phenomenon.^ Whether the subsystem retrieves (i.e. , 
reverses the role regression) every learner or not 
(it won't) the counseling at this point can be very 
valuable in helping learners and in compiling data ^ 
that can be used to construct profiies of learners 
that will help assess and modify all the subsystems. 
This is a function of formative evaluation. 

there may be a tendency as the systems are rou-^ 
tinized for a certain rigidity to creep in; for man- 
agers and staff people to shrug their shoulders over 
the non-start phenomenon because the learners who do 
progress through their roJLes keep everybody busy. 
Furthermore, the nature of most institutions — and the 
people who staff them--is to focus on what appear to 
be institutional successes rather than failures. 
(Non starts do not necessarily represent failure, 
but they may be categorized as failures by aa unknow- 
ing public or higher institutional authority unless 
the program people research the phenomenon, work aC; 
improvement, and have the evidence to avert a mis- ' 



reading of the raw, superficial statistics.) 
This, again, is a. function of formative evalua- 
tion. 

The same observations may: be made about 
learner branching in Role 6, persisting or not 
persisting in learning. Research has generally 
indicated that the learner who actually begins is 
more likely to persist; that a learner who per- 
sists through a third of a course is more likely 
to persist to the half-way mark; that a learner 
who passes the half-way mark is likely to complete 
the entire course, Jgain, there ate many causes 
for lack of persistefice, some completely outside 
the institution's responsibility* but some that 
may be directly the result of: problems that have 
their origin in the instructional system itself. 

Formative evaluation will probe learnei: pro- 
gress to monitor what is going on, sort out learners 
who cease for reasons external to the system, arid 
.those who are hung up for reasons that are the in- 
stitution's responsibility. Profiles of learner 
persistence in each course are instituted and main- 
tained; if a profile, for example, shows that a sub- 
stantial, proportion of learners in X course falter at 
Lesson 7, this is an unmistakable signal that some- 
thing unanticipated is happening at Lesson 7. What is 
it? Why? What must be done to correct the situation? 
Who is responsible for modifying the lesson? For coun- 
seling the learner? When must all this be done? 

Formative evaluation feeds' information to decision 
makers that will keep. the system in reasonable equili- 
brium. Note that the total system (the stages and sub- 
stages working together) are in effect a closed loop. 
Each successive stage is dependent for its success on 
the success of the preceding stage. For example, if 
there is a problem with Lesson 7 in a course, and sub- 
stantial attrition occurs at that point, the stages of 
the system that are to serve completing learners are 
endangered. Learners in sufficient numbers; to support 
the later stages do not get that far. Objectives 
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(both individual learner and social-institutional) 
are riot achieved;* the unit may appear to be over- 
staffed; per unit costs escalate; a budget crisis 
may arise; the program may come up for review by 
higher authorities. Formative evaluation is in- 
tended, to signal malfunctions early on, to pin- 
point causes, and offer evidence on which deci- 
"* - > - 

sions may be made for correction—before a crisis 
occurs* ; 

If there is no built in formative evaluation 

- _ - *. i - ' - - — 

system to be used as a guide in institutional de- 

velopment, the manager frequently does not know 

where to look when trouble occurs ♦ All he knows 

Is that learner and social-institutiorial goals are 

not. being met. In a complex system the trouble 

could be anywhere. A full blown comprehensive, 

* — _ _ - 

summative evaluation by a neutral outsider is^ 

time-consuming arid expensive * The patient may 
die before the right diagnosis and treatment can 
be determined. 

' 2.21. Stage III 
A subsystem that works with learners in Role 7, 
reaching or not reaching completion. Learners 
close to c6mpletion,Otpproaching final exams, wind- 
ing up the submission of all learning exercises, 
are likely to complete. But some do riot; and some 
who do complete^are failures as a result of a poor 
examination, poor overall achievement or other 
causes. Learner failure cannot be avoided. There 
are learners-who are lazy, who fail to show requi- 
site progress, who simply have failed to learn des- 
pite efforts made by others to help them achieve suc- 
cessfully. However, self selected learners who have 
persisted to cpmpletion have exhibited a high level 
of motivation plus a willingness to invest energy 
and resources over a period of time. It would seem 
unrealistic to classify them as lazy; or uncoopera- 
tive, or lacking in resourcefulness, or even lacking 
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/in capacity .at the end of the course, in order to 
& ' * * - - - 

explain failure. Such causes for failure should 

have shown up earlier, been noted so that counsel- 
ing and instruction could be modified accordingly, 
* ; in order that an unqualified learher does not; in 

fact approach the final stages unless he has,reason- 
ably good prospects for satisfactory completion. In * 
a well.»designed course the risk of failure should di- 
minish as the learner progresses even though it can 
. never :be completely eliminated. -Hence the final exa- 
minations ts as much a test of the instructional sy- 
stem as it is of the learner; \ * ^ >; 

Distant learners face one examination hazard that 
deserves careful study. .Throughout a course of instruc- 
tion, distant learners generally prepare their lessons 
p: in an open book, untimed -'format./. Tin is format works well, 

1/ • , * preserves freedom of choice; and is essential for non- 

captive learners. However, learners in this mode are 
not prepared for conventional examinations that are 
- supervised, timed and exclude aqcess. to materials. 

Some distant learners find, this switch of format dis- 
astrously upsetting. Yet for quality control and certi- 
fication-accreditation reasons, the institution may re- 
quire conventional examinations. The system must there- 
fore anticipate the problems and prepare the learner to 
meet them. A mid course examination may be needed not 
only to check progress,, but chiefly to give the learner 
experience in taking* a final examination in a different 

format. Lessons may be designed to. give practice in some 
V- - - r 

of the skills important ^ taking examinations. 

Formative evaluation of the system and subsystems 
will yield up .evidence of "what is actually going. on 11 
so that the managers as well as the learners are prepared 
k 1/ : * for eventualities and ready to make decisions that will 

pf: improve performance and success.. 

• But even successful achievement is not the end. Learn- 

ing is continuous throughout life. With completion pi any 
/ V part, a new cycle is ready to begin. The institution should 

//: // not abandon the successful learner anymbre than it abandons 
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learners who. have regressed in. various roles through- 
out the process. The subsystem must cope with learner 
and societal needs at the end of a process as well as 
at the beginning. Examination, certification, accredi-* 
tation, transfer, continuation, counseling towards new 
or continuing goals, employment — these are some of the 
activities that cluster when the Role 7 learner achieves' 
closure. Since a substantial proportion of the enroll- 
ments of educational institutions is composed of pre- 
viously successful learners (the more you learn, the 
more you want to learn) an adequate subsystem for cop- 
ing with successful ilearner needs is important to- the. 
* development of the institution. - 
The tentative three-stage model (system and subsystems) - suggested 
above is not yet complete. A system model is not complete unless 
it clearly indicates.: — 
What is to be done 
Why 

How the work is performed (the various systems 

and subsystems in stages) 
Who is responsible for what 

When things are scheduled to take place (sequence 

or order, including branching) 
' janisms for sensing what is actually happening 

(continuous feedback analysis through formative 

evaluation) 

Check points for monitoring and developmental deci- 
- s ion-makings 

Participatory relationships with learners, teachers, V 
counsellors, administration and others as neces- 
sary or desirable 
The institution builder who has progressed this far can now begin 
to flesh out his own model.. Fleshing out (supplying in detail all 
that the model calls for) is likely to be successful because of 
-the preliminary work already accomplished. (6) Bringing together 
the results of previous steps (conceptual matrix, systems develop- 
ment, analysis of learner roles, and tentative three-tftage model 
with subsystems) may be easier if a roc deling matrix is followed. 
Matrix for Mode ling a Correspondence Teaching and Learning Insti - 
tution . A matrix as a guide in fleshing out the model is suggested 
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2.25. When the planning model is fleshed out, each subsystem can be 

, modeled. It is desirable that the institution builder carry out 
this step to test the completeness, accuracy and viability of the 
total systems model. Following modeling, the systems and subsystems 
should be piloted, tried out. Here the sensing mechanisms are also 
tested, and the information yielded enables the institutional builder 
to modify and improve the model where needed. Full operation fol- 
lows the piloting and modification. 

2.26. The steps in institution building suggested here may seem more com- 
plex, burdensome and unwieldy than building the institution itself. 
Experience shows that they are not. Ordinarily the theory* and prac- 
tice of institution building is assimilated over a long period of 
time. Here a grand design has been compressed into a single presen- 
tation, chiefly as a vehicle to convey the place and role of forma- 
tive evaluation in institution building. 

Ends and Means < * 

Formative evaluation systems design, and all the other concepts and practices 
discussed, are not ends in themselves. They are only means towards those lar- 
ger institutional ends which animate social and institutional planners and 
builders. Their chief virtue is that they continually remind builders of the 
true ends of alL institution building— the improvement of the lives of indivi- 
duals in society, and thereby the improvement of society itself. 
3.1. Early on in this paper it was mentioned that in planning and build- 
ing one has to cope not only with what is present as a given in any 
* social or institutional situation, but also with what one has built 
to ameloriate, meet needs. The consequences of intervention (in an 
individual, institutional or social sense) are not always occurs tely 
foreseen, no matter how carefully the model has been designed and 
tested. Means which have been employed towards valid goals tend to 
become goals themselves; and valid goals when achieved tend to become 
means by which other goals may be realized. John Dewey (7) noted this 
most forcefully in pointing out, "In the continuous temporal process 
of organizing activities into a co-ordinated and co-ordinating untfty, 
a constituent activity is both an ends and a means: an end insofar 
as it is temporally and relatively a close; a means insofar as it pro- 
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vides a condition to be, taken into account in farther acti- 
vity. 11 He also reminds us that "No human activity operates in 
a vacuum: it acts in the world and has materials upon which 
and through which it produces results. . . .That organization 
which is the •final* value for each concrete situation of valua- 
tion thus forms a part of the existential conditions that have 
to be taken into account in further formation of desires and in- 
terests or valuations. 11 

In this paper formative evaluation has been suggested as a means 
which will aid in the building of an institution*; a process by 
which the manager can monitor the institutions development* and \ 
through which the consequences of intervention in the universe 
which comprises the institution may be more fully and accurately 
known. 

Ail this is also implied in "Murphy's Law" and its subsequent amen- 
dments. Murphy's Law expresses the comnon man's observation of 
how things work out in a complicated world: "If anything can go 
wrong, it will. 11 Apocryphal amendments to the law include, "If 
* anything can go wrong, it will, when and where you least expect itV; 
and, "If anything can go wrong, it will, not only when and where you 
least expect it, but you will be the last to know." The employment 
of formative evaluation as a function of institution building may 
help prevent the unwanted consequences which sometimes dog the acti 
vities 6f institution builders; and create an institution which ac- 
hieves its purposes. 
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APPENDIX 



Objective 
1 



Functlo a | Criteria 

l»4,7 | 1. Do courses selected and pro- 
duced meet the needs of: INCE 
participants? the nation? 



Measures 

1. enrollment statistics 
2* completion statistics 
3* quality indices 1 (level of partici- 
pant performance) 

4. participant follow up survey and 
profile (continuing) 

5. national needs and INCE policy* . 
6* field director survey (annuaiy 
7* participant employment indice 

(annual) 



2.3 



2, 3, 7 12. 



' hat problems in the develop* 
ment, use and application of 
methodology have been identi- 
fied and solved through re* 
search and experimental stu- 
dies? 



1* The identification of problems has 
been made a fixed responsibility of 
some person* 
2* Time is allotted for a person or pet 
sons to define problems, set ui for- 
mat studios, gafher data, formulate 
hypothetical solutions, test validit 
and apply solution* 
3* Ad hoc functional arrangements made 
with outside specialists for speci- 
fic research, experimental, or pro- 
blem solving activities* 
4* Research, experimental and problem 
solving studies have been reported 
at professional conferences, and/or 
have been published* 
5* INCE specialists in correspondence 
study are sought ^for consultation, 
advice, leadership in a larger spfier 
(regional, national, international)* 
6* Publications a*.J reports go beyond 
the descriptive to problem solving 
on a generalized and theoretical 
basis for broader application* . 
7* Problems identified and studied do 
not persist; or new studies are 
undertaken to achieve solution. 



ERLC 



3. Is the correspondence depart- 
ment administered efficiently 
in terms of its mission, ob- 
jectives, and functions? 



1. enrollment statistics 
2* completion statistics 
3* lesson load statistics 
4* participant quality indices 
5* collaboration with other related 
departments as needed for use of 
staff, resources 



I 6. regular reporting 



Criteria 



Measures 



! 7. Development/growth according to „ 
; planned and systematic strategy and 

i fc Implementation 

1 8* Productivity increasing proportional 
with development phases I, II, III* 

Cost of administration diminishing 
proportional with development phases 
I, II, III. 
10* Quality control measures in force 

and standardized* 
11. Comprehensive evaluation, in collabora 
tion with evaluation unit. at perio- 
dic intervals for decision making 
with respect to mission/objeiclives ■ 
and functional success/achieveinent. 



Does the correspondence in- I l # 
s true tion provided succeed 1 2. 
in: 

a) motivating participants? I 3. 

b) being relevant to partici-J 4. 
pant situation goals, as- 
pirations, problems? 

c) communicating information, J 5. 
process and content? 

d) satisfying the needs of ] 6. 
participants and national 
needs? 

e) maintaining a quality 
level at least as high as | 7. 
conventional instruction? 

f) employing methods, tech- J 8. 
niques, media and materi- 
als that reach partici- \ 9. 
pants effectively, ie., 
give easy access and pro- 
vide satisfactory feed- |l0. 

_back? 

g) ^ defining problems needing 111, 

study? 

h) mobilizing INCE resources 
from various units for 
resourceful development . 
of materials, processes, 
content? 

1) a r plying instructional 
know-how e f f ec tive ly ? 

j) applying results of parti- I 
pation in professional 
training and upgrading, 
continuing^education, in- 
dependent learning, and 
attendance at conferences 
and meetings of a profes- 
sional nature? 



enrollment statistics (cumulative) 
dropout, failure statistics (cumula- 
tive) 

review, inspection of all materials 
maintenance of problem file for 
correction of materials at points 
needed? Removal of obsolete elements 
Development/maintenance/revision „ on 
regular schedule. 

Materials, content, approach, design 
from participant situation and pro- 
file analyses. (Continuing and 
cumulative) 

Training and upgrading programs for 
instructors, others. 
Training, upgrading for course 
writers, developers. 
Training, upgrading for field repre- 
sentatives/director of centers, ad- 
visers to participants. 
Close study at periodic intervals of 
time factors in feedback loops. 
Spot checking at intervals of lessons 
taught by instructors for fairness, 
accuracy, completeness, positive and 
encouraging attitude, motivational 
relevance. 



Criteria 

Does the learning of the par- 
ticipants: 

a) meet INGE quality levels 
at least as well as con* 
ventional methods? 

b) satisfy practical needs of 
participants for entry in- 
to job marketplace or job - 
upward nobility? 

c) provide regular feedback . 
to instructors and course 

, developers/maintainers for 
the surveillance of over- 
all goals, methods, ma- 
terials, etc? 

d) carry to the mastery level 
for concepts, processes, 
information and skills 
that are essential to 
practical use and appli- 
cation in life? 

e) coordinate with other 
learnings obtained else- 
where so as to be rein- 
forcing and to contribute 
to continuing learning 
through whatever methods? 

*f) indicate adequate and suc- 
cessful use of communica- 
tions media and strong de- 
pendence upon sense learn- 
ing for development of 
ability to abstract and 
apply as necessary in job 
performance? 

g) occur through access sy- 
stems designed to minimize 
loss of time, reduce con- 
fusion, and emphasize _ 
practical usefulness of 
learnings? 

h) yield to analysis so that 
the range of individual 
successes and failures is 
clearly visible to the in- 
structor for individualiza- 
tion of instruction, dif- 
ferentiation of responses, 
and clear and justifiable 
identification of failure, 
average and exceptionally 
successful participants? 

i) enable exceptionally suc- 
cessful students to be 
singled out for congratu- 
lation? 

J)l permit- analysis for learner 
problems, leading to main- 
tenance, revision and fur- 



% Measures 

1. enrollment statistics 

2. drop-out, failure statistics 

3. exani nation/ lesson statistics 

4. compare quality level of partici- 
pant achievements with other pro- 
grams • 

5. Survey of standard feedback of 
participant attitude towards each 
course, with demographic data (for 
profile), problems, satisfac tion/dis 
satisfaction. 

6. Follow up survey (annual) of success 
of participant drop outs, failures — 
re: en try /change in job market; 
further learnings. Later re-assess- 
ment of course taken— add to profile 

7. Proportion of new enrollments that 
# are repeat -participants by own 

choice. 

8. Periodic (annual or bi-annual) sur- 
vey of field directors re: effective- 
ness of courses, " progress of studies 
etc • (anonymous) 

9. Periodic (annual or bi-annual) sur- 
vey of instructors, course deve- 
lopers of learning problems, effec- 
tiveness of process, materials and 
usefulness of course (anonymous) 

10. Identification of top 10% of partici- 
pants in success, and special recog- 
nition of, linked with profile, job 
entry and upgrading data. (Promotion 
public relations.) Suggestions for 
improvement. , 
111. Survey of instructor lessons for 

evidence of adaptation to individual 
differences. 
12. StuJy of lesson service — time/etc. 
13.. Maintenance of problem file for re- 
vis ion/develo pmen t . 

14. Definitions of problems in need of 
study. 

15. Evidence (attendance, writing, re- 
porting) of professional development 
gained- from attendance at conferences 
and meetings. 

16. Review panel for methods, materials. 



A ibiefctives 


Function 


Criteria 


Measures 






k) satisfy the expectations 


* 

* 






as well as needs of parti- 








cipants? 








1) reflect the application 








(in' methods, materials, 








processes) of knowledge of 








how learning occurs? 





5.8. . Summary of measures recommended : 



Time Frame 



1. 



2. 



3. 



Reports of 
Statistics: 



enrollment, current & cumulative 
completion, current & cumulative 
dropouts, current & cumulative' 
lesson loads, current & cumulative 
exam loads, current & cumulative 



satisfactory 
unsatisfac tory 
(failed) 



Course Statistics Analysis 
euro 1 Imen t 
completion I 
dropouts V from above 

lessons loads I 
exam loads ^ { J 

problems noted by participants 
problems noted by instructors 
problems noted by field directors 
problems noted by supervisors 



monthly, 
with semi- 
annual and 
annual sum- 
•nary, 
(for department, 
director, - publi- 
city) 



semi annual; 
or annual 

(for department, super- 
visors, course developers 
training and develop- 
ment) 



Methods/materials review panel (1 administrator-Chairman 

or . instructor 
one course writer/deve 

loper 
one participant 
one producer) 
(use of data compiled in #2; 
Recommendations? Actions on? 



Identification of problems requiring 

special study (ie., research; experimentation 

piloting, etc.) 



annual 
Review each course 
every 2 or 3 years; 
oftener if pro- 
blems occur; less 
frequent if course 
proceeds with few 
problems . Review 
Panel thus looks 
at 1/2 or 1/3 of 
courses annually. 



annual (for bud- 
ge t ing/ planning) 



9 

:RIC 



Field director survey 

special problems m - 

assess meeting needs of students 

suggestions for new courses, modification of old 



annual 



6. Participant Survey - continuing (part of each course enrollent; completed 

at time of final exam; anonymous) 

Probe: * problems 
attitudes 

satisfaction of needs 
suggestions for improvement 
practicality _ 7 • 

future plans for learning X a 
personal data 



(Report to department chairman) 



7. Participant Follow-up Survey 

(Probability sample only) 

a) successful completers 

b) unsuccessful completers 
(compare) 

c) dropouts (add items getting 
getting at why) 



repeat some of questions 
in survey 6. 

Present status: 
employed in area 
not employed in area 
employed in other area 



one study annually 
of completers, non- 
completers for im- 
provement of in- 
struction* courses, 
revisions, train- 
ing, etc, 
(ReportTfco 

department 

chairman) 



8. Course revision/development panel 



1 writer 

1 supervisor 

1 methods/ technology 

1 administrator (chairman) 



annual; act on 
information from 
other measurer; 
prepare report 
for planning, bud- 
geting, training 



9. Comparative Statistics Study 

comparison of enrollment, persistence, success and 
quality indices vis a vis correspondence instruc- 
tion and other types of instruction. Also (if 
^ known) disposition/ie., what happens to participant 
after course • ? .employed?) 
comparative unit costs (see #10) 

top 5%-10% of students by name for special congratu- 
lations • 



annual 

(for department 
chairman, public 
release) 



10 • Productivity Survey 

work output of department 

and individual members of department 
proportional cost (%) of administration 

teaching 

materials/other costs 



11. 



9 



Comprehensive Evaluation 

by Evaluation unit and correspondence instruction unit 



annual- 
cumulative com- 
parison for trends 

(for use in 
publicity; plan- 
ning and bud- 
geting for deve- 
loping decision 
making quality 
control 
compare #9. 

(to department 
chairman) 

every- three years 



12. Report of Feedback Time 

a) center participants 

b) mail, participants 



13* Profile of Participants— different levels 

different courses 

from continuing course survey data and 
follow-up data 



14* Instructor Survey (anonymous) 

for problems 
suggestions 
attitudes 
new course ideas 
etc* 



15. Lesson Review Panel 

(random sample 
review) 



1 administrator, chairman 

2 instructors ~ ~ 
1 field representative 

1 part-time or former participant 
* or professional 



annual 

to department chairman 

note problems identi- 
fied; action taken 



annual 

to department chairman 
for use in development; 
revision, planning, 
methodology, materials, 
training, reportQijg 

annual 

to department chairman 



for department chair- 
man 
training 

planning, development 



16. Periodic staff meetings to discuss any of above, 
reports of reading, conference, meetings, policy, 
training, etc. . . .or periodic staff bulletin to 
carry same function demonstrations. 

(By department chairman and collaboration 
of other specialists as needed, both* from 
within and without the department.) 



3 times/year with 
sufficient time to 
explore problems/ 
statistics, reports 
in depth, discuss, 
sense direction of 
training needed. 



Report of Assessment & Development , Dept. of Correspondence Instruction, INCE, 
Caracas. C. A. Wedemeyer, 1973, pages 48-53. ' 



